The Prague Bulletin of Mathematical Linguistics NUMBER ? ? ? JULY 2011 1 – 8 Metric combination for the Machine Translation optimisation tool MERT

The main metric used for SMT systems evaluation an optimisation is BLEU score but this metric is questioned about its relevance to human evaluation. Some other metrics already exist but none of them are in perfect harmony with human evaluation. On the other hand, most evaluations use multiple metrics (BLEU, TER, METEOR, etc.). Systems can optimise toward other metrics than BLEU. But optimisation with other metrics tends to decrease BLEU score. As Machine Translation evaluations still use BLEU as main metric, it is important to minimise the decrease of BLEU. We propose to optimise toward a metric combination like BLEUTER. This proposition includes two new open source scorers for MERT, the SMT optimisation tool. The first one is a TER scorer that allows us to optimise toward TER; the second one is a combination scorer. The latter one enables the combination of two or more metrics for the optimisation process. This paper also presents some experiments on the MERT optimisation in the Statistical Machine Translation system Moses with the TER and the BLEU metrics and some metric combinations. c © 2011 PBML. All rights reserved. Corresponding author: [email protected] Cite as: Christophe Servan, Holger Schwenk. Metric combination for the Machine Translation optimisation tool MERT. The Prague Bulletin of Mathematical Linguistics No. ???, 2011, pp. 1–8.